← All Tags
# local inference → llm → glm 5.2 → quantization → dynamic gguf

1 link tagged with all of: local inference + llm + glm 5.2 + quantization + dynamic gguf

Click any tag below to further narrow down your results

+ unsloth (1)

Links

GLM-5.2 - How to Run Locally | Unsloth Documentation

This guide shows how to run Z.ai’s open-source GLM-5.2 model on local hardware using Unsloth Dynamic GGUF quantizations. It covers memory requirements for 1-bit and 2-bit setups, recommended inference settings, and step-by-step instructions for Unsloth Studio and llama.cpp. The article also explains KLD benchmarks and quantization accuracy trade-offs.
  • GLM-5.2's 744B-parameter model can run on a 24GB GPU plus 256GB RAM using Unsloth's 2-bit dynamic quant (239GB, ~82% top-1 accuracy), an 84% size cut from the full 1.5TB model.
  • The 1-bit quant shrinks total memory needs to ~223GB but drops accuracy to ~76.2%.
  • 4-bit and 5-bit dynamic quants are nearly lossless per KLD testing, making them the better choice when accuracy on out-of-distribution tasks matters most.
  • Dynamic quantization keeps critical layers at higher precision while aggressively compressing less important ones, which is why even 1-2 bit versions stay broadly usable despite the accuracy trade-off.
Last saved Jun 24, 2026 · 7 min read
glm 5.2 + unsloth quantization local inference llm dynamic gguf